Free Lunch for Surgical Video Understanding by Distilling Self-supervisions

Self-supervised learning has witnessed great progress in vision and NLP; recently, it also attracted much attention to various medical imaging modalities such as X-ray, CT, MRI. Existing methods mostly focus on building new pretext self-supervision tasks reconstruction, orientation, masking identification according the properties of images. However, publicly available models are not fully exploited. In this paper, we present a powerful yet efficient framework for surgical video understanding. Our key insight is distill knowledge from trained large generic datasets (For example, released ImageNet by MoCo v2: https://github.com/facebookresearch/moco ) facilitate self-supervised videos. To end, first introduce semantic-preserving training scheme obtain our teacher model, which only contains semantics models, but can produce accurate data. Besides with contrastive learning, distillation objective transfer rich learned information model Extensive experiments two phase recognition benchmarks show that significantly improve performance existing methods. Notably, demonstrates compelling advantage under low-data regime. code at https://github.com/xmed-lab/DistillingSelf .